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Abstract 

This paper describes the interactive tutoring feedback model (ITF-model; Narciss, 
2006; 2008), and how it can be applied to the design and evaluation of feedback 
strategies for digital learning environments. The ITF-model conceptualizes 
formative tutoring feedback as a multidimensional instructional activity that aims 
at contributing to the regulation of a learning process in order to help learners 
acquire or improve the competencies needed to master learning tasks. It 
integrates findings from systems theory with recommendations of prior research 
on interactive instruction and elaborated feedback, on task analyses, on error 
analyses, and on tutoring techniques. Based on this multi-dimensional view of 
formative tutoring feedback methodological implications for designing and 
investigating multiple effects of feedback under multiple individual and situational 
conditions are described. Furthermore, the paper outlines how the implications of 
the ITF-model have been applied in several studies to the design and evaluation 
of tutoring feedback strategies for digital learning environments (e.g., Narciss, 
2004; Narciss & Fluth, 2006; Narciss, Schnaubert, Andres, Eichelmann, Goguadze, 
& Sosnovsky, 2013). 
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I. Introduction 

Feedback is considered one of the most powerful factors influencing learning in various instruc¬ 
tional contexts including digital learning environments (e.g., Hattie & Timperley, 2007; Hattie & 
Gan, 2011). In instructional contexts the term feedback refers to all post-response information 
which informs learners about their actual state of learning or performance in order to regulate the 
further process of learning in the direction of the learning standards strived for (Narciss, 2008; 
2012a; Shute, 2008). This notion of feedback can be traced back to early cybernetic views of 
feedback (e.g., Wiener, 1954), and implies that a core aim of feedback is to reduce gaps between 
current and desired states of learning (see also Ramaprasad, 1983; Sadler, 1989; Hattie 2009). 

Feedback can be provided by various external sources of information (i.e., teachers, peers, 
parents, computer-based trainings) in a large variety of ways, and by internal sources of 
information (i.e. information perceivable by the learner while task processing). Modern information 
technologies increase the range of feedback strategies that can be implemented in digital learning 
environments. 

The interactive potential of modern information technology makes it possible to provide learners 
not only with outcome feedback types but also with tutorial feedback strategies. Tutorial feedback 
strategies combine formative elaborated feedback with tutoring and mastery learning strategies: 
They provide formative evaluative feedback components that help learners to become aware of any 
gaps that exist between their desired and their current state of knowledge, understanding, or 
competencies. Additionally, they provide elaborated feedback components (e.g., hints, explana¬ 
tions, attribute-isolation examples) that are aimed at supporting learners in acquiring the 
knowledge and competencies necessary for mastering learning tasks. Thus, the focus of this 
elaborated information is on tutoring students to detect errors, overcome obstacles and apply more 
efficient strategies for solving learning tasks. In doing so tutorial feedback strategies offer 
strategically useful information for task completion without immediately providing the correct 
solution. Tutorial feedback strategies prompt the learner to use this strategic information to solve 
the learning task in a next trial. Furthermore, after successful task completion, they provide 
confirmatory positive feedback components (cf. Narciss, 2006; 2008; 2012a, 2012b; Narciss & 
Huth, 2006 describe in detail such a tutorial feedback strategy). 

However, research on intelligent tutoring systems illustrates that designing and evaluating tutorial 
feedback strategies for digital learning environments is a very challenging task (e.g., Arroyo, 
Woolf, & Beal, 2006; Arroyo, Woolf, Royer, Tai, Muldner, Burleson, & Cooper, 2011; Goldin, 
Koedinger, & Aleven, 2012; Mitrovic, Ohlsson & Barrow, 2013). Thus, until now, many digital 
educational systems do not provide tutorial feedback strategies but simple feedback strategies 
offering knowledge of result, (e.g., by merely stating or indicating if the response is correct or 
incorrect, or by flagging errors), and/or knowledge of the correct response. To provide an empirical 
and theoretical basis for the design and evaluation of tutorial feedback strategies, Narciss (2006, 
2008, 2012a, 2012b) has developed a multidimensional feedback model, hereafter referred to as 
the Interactive tutoring feedback model (ITF-model). 

The purposes of this paper are first to describe the ITF-model and its implications for the design 
and investigation of tutoring feedback strategies. Second, studies and findings from my own 
research team (e.g., Narciss, 2004; Narciss, & Huth, 2006; Strijbos, Narciss & Duennebier, 2010; 
Narciss, Schnaubert, Andres, Eichelmann, Goguadze, & Sosnovsky, 2013) will be summarized to 
illustrate how the implications derived from this model can be applied to the design and evaluation 
of tutorial feedback strategies for digital learning environments. Finally, implications for further 
research and instructional design will be discussed. 
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II. The Interactive Tutoring Feedback Model 

Recent feedback frameworks consider a learner as an active constructor of knowledge, and thus 
claim that the most important function of feedback is to provide learners with formative or tutoring 
information on their current state of learning which help them to regulate their learning process 
successfully (e.g., Butler & Winne 1995; Hattie & Timperley, 2007; Narciss, 2008; Nicol & 
Macfarlane-Dick, 2006; Hattie & Gan, 2011; Shute 2008). 

Based on this view of feedback, Narciss (2006; 2008) has synthesized the theoretical and empirical 
insights of feedback frameworks and research (e.g., Butler & Winne, 1995; Hattie & Timperley, 
2007; Kluger & DeNisi, 1996; Kulhavy & Stock, 1989; Mory, 2004) as well as research on 
formative assessment (e.g., Black & Wiliam, 1998; Sadler, 1989) by the Interactive tutoring 
feedback model. This ITF-model conceptualizes feedback as a multidimensional instructional 
activity that aims at contributing to the regulation of a learning process in such a way that learners 
acquire the knowledge and competencies needed to master learning tasks. 

Conceptualizing feedback as an instructional activity that aims at contributing to the regulation of a 
learning process makes it possible to use the core insights provided by instructional research to 
analyze possible factors and effects of (tutoring) feedback strategies (see also Hattie & Gan, 2011; 
Hattie, 2009). Instructional models and research suggest that the effects an instructional activity 
can have are determined by the quality of the instructional activity (e.g. scope, nature, and 
structure of the information provided, and the form of presentation), individual learning conditions 
(e.g. prior knowledge or level of competencies, meta-cognitive strategies, motivational dispositions 
and strategies), as well as situational conditions of the instructional setting (e.g., instructional 
goals, learning content and tasks). The ITF-model links these issues with insights on feedback from 
systems theory (e.g., Ramaprasad, 1983) while, on the other hand, aims at integrating findings 
from systems theory with recommendations of prior research on formative assessment and 
feedback (e.g., Black & Wiliam, 1998; 2009; Nicol & Macfarlane-Dick, 2006; Sadler, 1989; Shute 
2008), on self-regulated learning and feedback (e.g., Butler & Winne, 1995, Winne & Hadwin, 
1998), on task analyses (cf. Jonassen, Tessmer, & Hannum, 1999), on error analyses (e.g., 
VanLehn, 1990), and on tutoring techniques (e.g., McKendree, 1990; Merrill, Reiser, Ranney, & 
Trafton, 1992; VanLehn, 2011). 

Based on regulatory paradigms from systems theory, the ITF-model views feedback as one of 
several basic components of a generic feedback loop. However, when regulatory paradigms from 
systems theory are applied to an instructional context, in which learners are provided with 
feedback by an external feedback source (e.g., teacher, peer-student or digital instructional 
medium), two interacting feedback loops must be considered (see Figure 1): the learner's feedback 
loop and the feedback loop of the external feedback source. 

The ITF-model describes the interaction among these two loops as follows: 

The controlled process for the two interacting feedback loops consists of the acquisition of 
competencies (i.e., conceptual and procedural knowledge, as well as cognitive, meta-cognitive and 
motivational strategies and skills), necessary to master the demands associated with learning 
tasks. Building on models of self-regulated learning (e.g., Butler & Winne, 1995; Boekarts, 1996; 
Winne & Hadwin, 1998) and on current multidimensional notions of competencies (e.g., Klieme, 
Leutner, & Kenk, 2010), the ITFL-framework assumes that the acquisition of competencies has to 
be controlled on several levels, namely the cognitive, motivational, and meta-cognitive level. To 
determine the controlled variables in a learning process with regard to these control levels it is 
necessary to identify and specify the cognitive, motivational, and meta-cognitive variables which 
are relevant for acquiring the desired competencies. 
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Figure 1: Components of the Interactive tutoring feedback model (ITF-model; translated and modified from Narciss, 2006, p. 70) 
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To regulate and control the acquisition of competencies the following elements of the internal and 
external feedback loops are needed: 

First, for each of the controlled variables a standard or reference level has to be determined. The 
ITF-model assumes that in the learner's loop Internal standard or reference levels are determined 
by the learner's subjective representation of competencies and the related standards. Individual 
learner factors such as learner's goals and prior level of competencies influence these subjective 
representations of competencies. In a similar manner the external standard or reference levels are 
determined by the external representation of competencies, and the latter are influenced by 
situational features of an instructional context, particularly by instructional goals or standards, 
approaches, content and tasks. If a teacher or peer serves as an external source of feedback their 
respective level of expertise or competencies has to also be considered as a factor influencing the 
external representation of competencies. 

Second, internal and external feedback loops require that the actual state of competencies is 
(continuously) assessed by both the learner (internal assessment) and also by the external 
feedback source (external assessment). These assessments result in an internal and an external 
feedback of the actual state of the learner's competency. 

Third, the internal and the external feedback can only be used to regulate the process of 
competence acquisition if in both loops there exist a controlling information processing unit (i.e., 
internal and external controller in cybernetic terms). In both controllers the feedback on the cur¬ 
rent state of competencies has to be compared to the respective desired level of competencies in 
order to evaluate to what extent the desired level of competence has been achieved. Based on this 
comparison, the external controller has to generate an external feedback message. If there is no 
discrepancy between the desired and the current competency state, this external feedback 
message may simply consist of appraising or confirming that the desired competency state has 
been achieved. If a gap between the desired and the current level of competencies has been 
detected, this external feedback message may provide evaluative information pertaining to this 
gap, as well as tutoring information (i.e., all kinds of suggestions for control actions that help close 
the gap). It is important to emphasize here that there is a very large variety of external feedback 
messages that can be generated by an external feedback source, and that just delivering the result 
of the external assessment in terms of evaluative information (e.g., grades; ranks) is only one 
possibility. 

Fourth, the external feedback message has to be processed in the learner's internal controller 
along with the internal feedback. This means that learners have to do several comparisons before 
they can generate their suggestions for control actions, and select the most appropriate one: They 
have to compare (a) their desired and their internally assessed current state of competencies 
( = internal feedback), (b) their desired level of competencies with the external feedback message, 
and (c) their internal feedback with the external feedback message. From these processes the 
learner has to generate an internal control action. To do so, the learner has to identify the 
source(s) of any discrepancies that are detected between the various internal and external values. 
Such discrepancies can occur when, for instance, internal or external assessments are inaccurate, 
the quality of internal or external feedback is poor, or the subjective task representation is 
incorrect or imprecise and thus leads to incorrect reference values. Consequently, the external 
feedback message may confirm or complement the internal feedback, or it may be discrepant to 
the internal feedback. Based on the results of these comparisons, learners have to generate their 
ideas or suggestions regarding one or several control actions that may help them to proceed in the 
direction of the desired level of competency. 

Fifth, learners have to select and transmit the internal control action(s) to the controlled process, 
where these control action(s) (e.g., error correction strategies; revision activities) have to be 
implemented. If learners have selected adequate control action(s), their controlled process, that is, 
their competence acquisition process should be improved. This improvement should result in a 
reduction of the discrepancy between the desired and the actualized state of competency. 
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III. Factors affecting feedback efficiency 

The assumptions of the ITF-model outlined above lead to the conclusion that the efficient 
regulation of competence acquisition with external feedback may be affected by factors of both the 
learner ( = internal loop), and the external feedback source ( = external loop). Both feedback loops 
contribute to the regulation of the same controlled process which is characterized by the 
competencies necessary to accomplish the requirements of the learning task(s). Thus, the require¬ 
ments of learning tasks and the related competencies constitute a third group of factors influencing 
the effects of a feedback strategy. 

a. Requirements of learning tasks and competence acquisition 

As outlined above, the starting point for both feedback loops is the process of acquiring the 
competencies necessary to master learning tasks in a given instructional context. This process can 
be more or less complex depending on the requirements of the learning tasks and the instructional 
objectives. In order to regulate a system successfully, it is crucial to describe its controlled process 
carefully and precisely. Describing the process of competence acquisition involves initially analyzing 
exactly what requirements are associated with the learning tasks, and what competencies are 
needed to master these requirements. Moreover, in order to generate control actions for the 
regulation of the process of competence acquisition, the errors and difficulties that could arise in 
connection with mastering task requirements must also be identified, as well as the information 
and strategies that are needed to overcome these errors or difficulties. 

Bloom's revised version of the taxonomy of learning objectives may serve as a basis for analyzing 
task requirements (cf. Anderson & Krathwohl, 2001). Analyzing learning task requirements on the 
basis of this taxonomy makes it evident that it is more difficult to precisely identify the content- 
related, cognitive, meta-cognitive, and/or motivational requirements for complex tasks (i.e. require 
higher-order knowledge and operations) than for simple tasks. Hence, identifying and precisely 
describing the underlying competencies is more difficult for complex than for simple tasks. Yet, a 
precise description of competencies provides the basis for (a) identifying the relevant control 
variables and their respective standards, (b) assessing the current state of these control variables, 
and (c) generating control actions. In other words, if a precise description of competencies is mis¬ 
sing, a learner and also an external feedback source (e.g., teacher, peer, instructional medium) 
cannot know which competencies are needed to master task requirements, and thus can neither 
reliably assess them, nor generate feedback or actions for improving these competencies. When 
designing formative feedback strategies for learning environments it is thus crucial to conduct 
thorough task analyzes in order to describe the competencies necessary for mastering the tasks. 

b. Internal loop factors - learner factors 

The ITF-model suggests that the quality of processing in the internal loop (= learner's loop) 
depends on several learner factors: 

• Learner's representation of standards, competencies, and task requirements : learners have 
to understand and represent task requirements and the related competencies. Whether 
learners are able to represent task requirements adequately and precisely depends on the 
complexity of these requirements, but also on individual factors such as learner's prior level 
of competencies (i.e., knowledge, meta-cognitive knowledge and strategies, motivation). 

• Learner's self-assessment skills: Learners have to monitor and assess their process of 
learning or competence acquisition in order to generate their internal feedback. This internal 
feedback can be compared to the learner's desired level of competence, and to the external 
feedback. It may thus serve, at least to some extent, as a frame of reference for the 
external feedback. Hence, learner's monitoring or self-assessment skills have to be 
considered an important factor for the processing of the external feedback. 

• Learner's skills and strategies in information processing: Learners have to process the 
internal and external feedback and compare them to their desired level of competencies, and 
generate control actions. Thus, a further learner factor identified by several reviews and 
meta-analyses of feedback research is learner's proficiency in mindfully processing and 
integrating information from several sources. 


S.Narciss 

Digital Education Review - Number 23, June 2013- http://greav.ub.edu/der/ 


12 


Designing and evaluating tutoring feedback strategies for digital learning environments on the basis of the interactive tutoring feedback model 


• Learner's will and skills in overcoming errors and obstacles. As shown in studies on feedback 
seeking, even the most sophisticated feedback is useless if learners do not attend to it (e.g., 
Aleven, Stahl, Schworm, Fischer, & Wallace, 2003; Narciss, Koerndle, Reimann, Mueller 
2004) or are not willing to invest time and effort in error correction. Aside from will, students 
also need the skills necessary for accomplishing the requirements related to error correction. 
Butler and Winne (1995) derived six maladaptive ways of feedback seeking and processing 
from Chinn and Brewer's (1993) work on how misconceptions may hinder conceptual 
change: Students may (a) ignore the external feedback, (b) reject the external feedback, (c) 
judge the external feedback irrelevant, (d) consider external and internal feedback as 
unrelated, (e) reinterpret external feedback in order to make it conform to the internal 
feedback, (f) make superficial rather than fundamental changes to their knowledge and/or 
beliefs. In all these cases the effect of the external feedback will be small. 

Note that all these internal factors may promote or constrain how well learners will be able to 
improve their competencies in the direction of the desired standards if they are provided with 
formative tutoring feedback messages by an external feedback source. Thus, the ITF-model 
suggests taking these learner factors into account when designing and investigating feedback 
strategies for (digital) learning environments. 

c. External loop factors - factors of the external feedback source 

The ITF-model also attracts attention to the following external loop factors which might contribute 
to the efficiency of both feedback loops: 

•Quality of external representation of standards, competencies, and task requirements: As in 
the internal loop, the starting point for most processes in the external loop is a precise 
identification and description of the competencies necessary for mastering learning task 
requirements. Based on this description, an external representation of these competencies 
and requirements, as well as the external level of standards has to be determined by the 
external feedback source. In the case of a human external feedback source (e.g., a 
teacher or peer), the quality of this external representation and definition of standards 
may vary depending on the level of (instructional) expertise of this external feedback 
source. In a digital learning environment, the external feedback can also be provided by a 
digital device (e.g., pedagogical agent, computer-driven feedback). In this case, technical 
characteristics as well as the level of instructional design expertise, on which the design of 
this digital system has been based, will influence the quality of the external representation 
of standards and competencies. For complex tasks in particular, a precise representation 
of what constitutes a high level of competencies in mastering the task requirements may 
be difficult for teachers, peers or instructional designers (Sadler, 1989). 

•Quality of external assessment: The accurate and reliable assessment of a learner's current 
level of competence is a necessary precondition for making meaningful comparisons 
between the desired and the current levels of competencies. To do so the external 
feedback sources have to work out which indicators are appropriate for measuring 
different levels of competencies in a valid and reliable way. 

•Quality of external data processing: Making meaningful comparisons between the assessed 
level of competence and the desired level of competence also depends on how well the 
external feedback source processes the information at hand. 

•Quality of design and communication of external feedback: A core factor influencing the 
power of a feedback strategy is the quality of the feedback message generated by the 
external feedback source. For example, if a discrepancy between the desired and the 
current level of competence is detected the external feedback source has to generate 
suggestions for control actions that may reduce this discrepancy. Key issues here are, how 
well does the external feedback source (a) understand why learners did not master the 
task requirements at the desired level, (b) identify which external feedback information 
may be provided to the learners in order to help them acquire the lacking competences or 
improve their week competences, and (c) communicate this external feedback to the 
learners in such a way that learners will attend to it and process it mindfully. 
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IV. Implications for the design of (tutoring) feedback strategies 

The ITF-model implies that researchers and designers have to adopt a multidimensional view of 
feedback in order to design and investigate formative feedback strategies, in particular formative 
tutoring feedback strategies with complex elaborated feedback messages (Narciss, 2008; see also 
Narciss & Huth, 2004; Shute 2008; see also Hattie & Timperely, 2007; Hattie & Gan, 2011; 
Thurlings, Vermeulen, Bastiaens, & Stijnen, 2013). 

This multidimensional view includes considering at least three facets of a feedback message to 
determine the nature and quality of the feedback message (Narciss & Huth, 2006, Narciss 2012b): 

Feedback scope and functions: Feedback can affect the learning process at various levels, and can 
therefore have numerous different functions. It can for example acknowledge, confirm, or reinforce 
correct responses or high quality learning outcomes, and in doing so promote the acquisition of the 
knowledge and cognitive operations necessary for accomplishing learning tasks (e.g., Mitrovic, 
Ohlsson, & Barrow, 2013). Feedback can also contribute to correcting errors, misconceptions, or 
inadequate task processing strategies. Moreover, it can prompt the application of metacognitive 
strategies (e.g., Butler & Winne, 1995; Mathan & Koedinger, 2005), or encourage students in 
maintaining their effort and persistence (e.g., Hoska, 1993; Narciss, 2004; 2008). Based on the 
models of "good information processors" (Pressley 1986), "intelligent novices" (Mathan & 
Koedinger, 2005), self-regulated learning (Boekarts, 1996; Winne & Hadwin, 1998), as well as 
Kluger and DeNisi's feedback intervention theory (Kluger & DeNisi, 1996), Narciss (2006, 2008) 
suggests classifying the widely varying feedback functions as cognitive, meta-cognitive, and 
motivational (see fig. 2). 

Feedback content: The large and quickly growing body of feedback research reveals that the above 
mentioned feedback functions have been addresses by a large variety of feedback types and 
strategies, which also vary widely regarding their content (see Hattie and Timperley, 2007; Mory, 
2004, Narciss, 2008; Shute, 2008; Thurlings et al., 2013, for recent reviews). Rooted in different 
instructional approaches, instructional designers and researchers have for example investigated the 
conditions and effects of the following feedback types: 

• knowledge of performance, providing learners with a summative feedback after they have 
responded to a set of tasks or accomplished a complex assignment (e.g., percentage of 
correctly solved tasks; number of errors; grade) 

• knowledge of result, providing learners with information on the correctness or quality of 
their actual response or outcome (e.g., correct/incorrect; flagging errors; good job) 

• knowledge of the correct response, providing the correct response or a sample solution to a 
given task 

• elaborated feedback, providing additional information besides knowledge of result or know¬ 
ledge of the correct response (e.g., hints, guiding questions, explanations, worked 
examples). Since there is a variety of elaborated information that might be added to know¬ 
ledge of result, Narciss (2008) suggests using at least five different categories in order to 
make a more subtle distinction regarding feedback types with elaborated feedback content: 

o knowledge about task constraints, offering information on task rules, task constraints 
and/or task requirements 

o knowledge about concepts, addressing conceptual knowledge by providing for example 
response-contingent hints on concept attributes, or attribute-isolation examples 

o knowledge about mistakes, offering information on errors or mistakes (e.g., flagging 
location of errors; or providing hints on error type or error sources) 

o knowledge about how to process the task, addressing procedural knowledge (e.g., 
task-contingent hints about procedural skills or problem solving strategies) 
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o knowledge about meta-cognition, eliciting meta-cognitive knowledge and strategies 
necessary for self-regulating the learning process (e.g., topic-contingent hints about 
useful sources of information). 

The findings of studies comparing several feedback types differing in their content are rather 
mixed. Some studies demonstrate for example positive effects of elaborated feedback types, while 
others show no effect or even negative feedback effects (for further details see the reviews by 
Kluger & DeNisi, 1986; Mory, 2004; Narciss, 2008; Shute, 2008; Thurlings et al. 2013). 

Feedback presentation: Feedback types also vary widely in the way they present or communicate 
the feedback content to the learner. Formal and technical issues related to the presentation of the 
feedback content constitute another facet of feedback instructional designers or all other types of 
external feedback sources have to cope with. In order to present for example the feedback 
content, it is necessary to determine the (a) feedback timing (e.g., immediate, delayed), (b) 
feedback scheduling (e.g., single try, multiple try, answer until correct), (c) codes and modes of 
feedback representation and delivery, and (d) adaptation strategy (e.g., non-adaptive, adaptive, 
adaptable, mixed-initiative or shared-control adaptation). 

Figure 2 summarizes the design implications of the ITF-model and reveals that in order to design a 
tutoring feedback strategy, instructional designers and researchers first have to develop an 
adequate understanding and representation of the relevant competencies, and determine the 
desired standards for these competencies. To do so, they have to analyze the conditions of the 
instructional context, for example, by conducting cognitive task analyses. Second, they have to 
analyze the conditions and characteristics of the learner or more generally, the feedback receiver, 
as well as of the available feedback source(s). Based on these analyses, instructional designers and 
researchers may design the tutoring feedback strategy by developing a coordinated plan which 
integrates clear and decisive statements regarding the following issues (Narciss, 2012): 

(a) What facet(s) of competencies should the feedback help to improve ( feedback scope and 
functions )? 

(b) Given the selected functions, what external post-response information should the feedback 
provide ( feedback content )? 

(c) When and how should the selected feedback content be conveyed to the learner ( feedback 
timing and presentation )? 

Unfortunately, the large body of feedback research provides either rather inconsistent or not 
enough findings concerning the effects of feedback conditions and facets (for further details see 
Narciss, 2008). Flowever, several authors have derived prescriptive principles for designing 
formative tutoring feedback strategies from recent feedback and formative assessment frameworks 
(e.g., Narciss & Fluth, 2004; Narciss, 2008; 2012b; Nicol & MacFarlane-Dick, 2006; Shute, 2008). 
Narciss (2012b, p. 1292) summarizes these prescriptive principles as follows: 
"Good feedback strategies should empower students as self-regulated and productive lifelong 
learners. Consequently, they should be rather interactive than just transmission focused and 
include the following steps: First, communicate the criteria and standards of high quality task 
processing and completion in order to help students develop an adequate representation of how to 
accomplish successfully the task requirements related to a high level of performance. Second, 
initiate self-assessment with regard to the relevant assessment standards and make self- 
assessment overt. In other words, help students to generate and use internal feedback which 
elicits the criteria they have met as well as those they have not (yet) met. Third, indicate the 
externally assessed level of performance and relate it to both, the standards aimed for and the 
self-assessed level of performance. Fourth, initiate reflection on corrective actions for closing gaps 
between the current level of performance and the standards. Fifth, provide tutoring information 
(i.e., hints, guiding questions, explanations, analogies) to help students select and apply adequate 
corrective actions if they fail to do so without assistance. Sixth, offer occasions for engaging in 
corrective actions which apply the tutoring feedback information (i.e., further attempts to 
accomplish task requirements) in order to improve learning and the level of performance. Seventh, 
elicit progress and emphasize successful attainments of high quality standards." 
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Figure 2: Conditions and factors influencing feedback processing and effects (modified from Narciss, 2012) 
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V. Implications for the evaluation of (tutoring) feedback strategies 

The ITF-model suggests that the effects of a (formative tutoring) feedback strategy can be 
manifold depending on the properties of the feedback strategy, the various conditions and factors 
of the feedback receiver (i.e., learner), the feedback source, and the instructional context. In order 
to investigate the effects of various types of external feedback strategies, researchers are thus 
faced with the following challenges (Narciss, 2006): 

• Multiple design varieties: Feedback messages and strategies, in particular, formative 
tutoring feedback strategies are multi-dimensional, and can thus be designed in many 
different ways. This large number of design varieties for external feedback strategies is 
reflected in a growing number of labels used to refer to the feedback messages or 
strategies investigated. Since new labels do not always refer to new kinds of feedback 
messages or strategies, it is very difficult to compare the findings of feedback studies. 
Thus, a first important step in evaluating the effects of feedback strategies is to describe 
accurately the content-related, formal and functional differences between the feedback 
messages or strategies under investigation. In terms of the various feedback dimensions 
and facets described above a detailed description of a feedback strategy also includes that 
the instructional setting under investigation must be carefully described, particularly its 
specific goals, typical learning tasks, and the competencies underlying these learning tasks. 
Moreover, the intended functions, the content of the selected feedback components, and 
the mode and form of feedback presentation must be described. Furthermore, it is 
necessary to indicate whether, and if applicable how instructional goals and feedback, 
typical errors and feedback, and individual learning level and feedback are tailored to each 
other. 


Feedback functions 

Anticipated effects 

Indicators for ant 

Observable 

licipated effects 

Reportable, 

i.e. assessed by items/scales 

Cognitive level 

• inform 

• complete 

• correct 

• specify 

• restructure 

• Recognize errors 

• Acquire lacking knowledge 

• Correct incorrect knowledge 

• Specify inaccurate knowledge 

• Correct incorrect associations 

• Number of errors corrected 

• Number of trials until correct 

• Number of task types for which 
mastery is achieved 

• Number of tasks the learner 
needed to achieve mastery 

• Information retrieval 

• Specific task processing steps 

• Specific strategies 

• Conditions under which strategies 
are applied 

• Response certitude 

• Assessment of number of errors 

• Assessment of feedback's 
usefulness 

Metacognitive level 

• Informing 

• Completing 

• Correcting 

• Guiding 

• Recognize incorrect strategies 

• Acquire missing strategies 

• Correct incorrect strategies 

• Attract attention to strategies 

Motivational level 

• Increase incentive 

• Decrease task difficulty 

• Increase probability of 
success 

• Associate success to 
effort 

• Increase probability of 
positive perceptions of 
competence 

• Task values increase 

• Task engagement increases 

• Expectations of success increase 

• Self-efficacy increases 

• Attribution patterns change 

• Effort increases 

• Perseveration increases 

• Positive self assessments are 
more likely 

• Number of tasks processed 

• Amount of time spent on 

- task processing 

- feedback processing 

• Number of tasks learners gave up 

• Number of tasks skipped 

• Assessment of task values 

• Assessment of task enjoyment 

• Assessment of willingness to 
task engagement 

• Assessment of task difficulty 

• Attributions 

• Expectations (e.g., self efficacy) 

• Perceived performance level 

• Degree of satisfaction with 
performance 


Table 1: Overview on indicators for investigating cognitive, meta-cognitive, and motivational feedback effects 
(adopted from Narciss, 2006, p. 93) 
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• Multiple effect levels and functions: Effects of external feedback messages or strategies 
may occur on several levels of the process of competence acquisition, because during 
competence acquisition, feedback may not only promote changes in observable outcomes, 
but also in the cognitive, meta-cognitive, and motivational skills and strategies necessary 
to achieve these outcomes. Furthermore, these different effects may have a cumulative, 
complementary, or interactive impact on learning outcomes (e.g., level of knowledge, 
meta-cognitive and motivational skills, or transfer performance; see for example Goodman, 
Wood, & Chen, 2011). This in turn means that investigating the effects of external 
feedback necessitates not only global performance metrics but also specific indicators for 
the measurement of differential effects on the various effect levels and outcomes. Table 1 
illustrates how such indicators can be worked. 

• Multiple effect times and periods: Closely related to the issue of the effect levels and 
functions of feedback are the issues of when and how long the feedback messages or 
strategies have an impact on competency acquisition. External feedback can contribute to 
changes that occur (a) during the treatment of current problem performance (e.g., reduce 
the time or number of trials to achieve mastery; see for example Mitrovic et al. 2013; 
Narciss & Huth, 2006), (b) after a short delay (e.g., improve next problem performance, 
support performance in an immediate post-test), or (c) after a far delay (e.g., retention 
and/or transfer of competencies). Thus, a complete picture of the effects of various 
feedback strategies can be obtained only if researchers use complex designs including data 
collected during and after the treatment. 

• Multiple feedback conditions: The ITF-model reveals that the effects of an external 
feedback message or feedback strategy do not occur in general, but emerge differentially 
depending on the various conditions and factors of the feedback receiver (i.e., learner), the 
feedback source, and the instructional context. For example, given the factors and 
conditions outlined above, the amount of time it takes for a student to correct an error with 
the help a tutoring feedback message is influenced by (a) the type of error and the sources 
of these errors, which originate in the type, complexity, and difficulty of the tasks and their 
underlying competencies, (b) the individual characteristics of the learner; (c) the quality of 
the tutorial feedback strategy; and (d) if and how the learner uses the feedback for 
correcting these errors. Consequently, studies on tutoring feedback strategies should either 
control the relevant feedback conditions or even investigate if they have moderating or 
mediating effects. 

• Multiple ways of feedback processing: The effects of various feedback strategies largely 
depend on if and how learners process and interpret the information provided. In addition 
to the cognitive factors of the learner (e.g. prior knowledge, strategic knowledge), above 
all, individual motivational factors such as self-efficacy and perceived task values, and 
individual meta-cognitive factors such as monitoring competencies and strategies play a 
role here. Flence, in order to draw differentiated conclusions about the effects of various 
types of feedback, not only cognitive but also individual motivational and meta-cognitive 
factors and the nature of students' feedback processing are worth being controlled. 

• Variability of error rate: External tutoring feedback messages can only be effective if the 
students have occasions to receive them, that is, if they make errors, encounter obstacles, 
or experience uncertainty during task processing. If students are highly proficient and 
confident about their proficiency, they will make few or no errors, and thus have only few 
occasions to benefit, for example, from elaborated tutoring feedback messages. Flence, the 
individual error rate determines how often students are provided with feedback, in 
particular with tutoring feedback messages. To cope with this problem, several researchers 
have selected students with a high error rate for participating in their feedback studies and 
found that elaborated formative feedback components were more effective than simple 
outcome feedback (e.g., Collins, Carnine, & Gersten, 1987, Salas & Dickinson 1990, 
Narciss & Fluth 2006). 

• Variability of error types: External tutoring feedback messages are considered to be most 
effective if their content is tailored to task requirements and competencies, or even to 
systematic errors. Flowever, in complex tasks a variety of different errors may occur (e.g., 
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more or less severe errors, more or less systematic errors) and the sources of errors may 
vary (see for example Eichelmann, Narciss, Schnaubert, & Melis, 2012). Moreover, 
VanLehn's work on bug-migration reveals that, if students encounter an impasse (i.e., fail 
with regard to a difficult task requirement), the errors, which they make due to this 
impasse, may vary from trial to trial (VanLehn, 1990; see also Wittmann, 2013). 

These challenges reveal that investigating the benefits of tutoring feedback strategies is a very 
difficult task, which can hardly be accomplished with simple experimental designs (see also Price, 
Handley, Millar, & O'Donovan, 2010). 


VI. Design and research based on the Interactive tutoring feedback model 

My research team has applied and is currently applying the implications from the ITF-model both to 
the design and investigation of system-provided tutoring feedback strategies (e.g., Narciss, 2004; 
Narciss, & Huth, 2006; Schnaubert, Andres, Narciss, Eichelmann, Goguadze, & Melis, 2011), as 
well as to the investigation of feedback strategies provided by teachers or peer-students (Strijbos, 
Narciss & Duennebier, 2010). The following sections summarize the methodological strategies, 
insights and findings of some of these studies. 

a. Tutoring feedback for concept formation tasks (Narciss, 2004) 

The ITF-model illuminates that individual and situational feedback conditions may influence if and 
how tutoring feedback strategies may be differentially beneficial. As mentioned above, it is thus 
necessary to control the relevant feedback conditions or examine if they moderate or mediate the 
effects of the feedback strategy. Thus, the two studies reported in Narciss (2004) aimed to 
investigate if and how, as well as, under what situational conditions learners' motivational charac¬ 
teristics (we focused on self-efficacy) influence the impact of tutoring feedback on learners' 
achievement and motivation when solving concept formation tasks. 

More specifically, these studies examined the assumption that, dependent on learner's self- 
efficacy, there are differential effects of tutoring feedback strategies on achievement and 
motivation (namely, task engagement, effort, persistence, and satisfaction with performance). This 
assumption was derived from theories and research on motivation suggesting that students' self¬ 
perceptions of competence (e.g., academic self-efficacy, academic self-concept) affect subsequent 
motivation in an activity (Bandura, 1977, 1997; Heckhausen, 1991; Wigfield & Eccles, 2000; 
2002 ). 

To investigate this assumption we used Bruner's concept formation task (Bruner, Goodnow & 
Austin, 1956) for the following reasons: First, this experimental concept formation task was novel 
to learners and thus, it offered enough occasions for providing the tutoring feedback. Second, prior 
research using this task provides empirical insights into the cognitive requirements and typical 
errors of this task. Based on these insights we developed the content of the tutoring feedback 
hints. Third, this concept formation task uses subject-independent stimulus-material, and thus 
affords the ability to tackle the problem of subject-related motivational differences (i.e., high vs. 
low interest) that may interfere or mask the effects of the tutoring feedback. 

In two experiments, students differing in their self efficacy for such concept formation tasks 
worked on a set of concept identification tasks. In cases of incorrect hypothesis about the concept, 
they received feedback according to their experimental conditions. These experimental feedback 
conditions were designed by combining several feedback components with a multiple-try strategy 
offering subsequent tries to the given item after incorrect hypothesis: The first feedback 
component provided outcome feedback (KR); subjects were told if their hypothesis about the 
concept was right or wrong. The second feedback condition consisted of outcome feedback and 
information on the location of mistakes (e.g., card 5 does not fit your hypothesis). The third 
feedback condition contained outcome feedback, location of mistakes, and a hint on how to 
proceed (i.e., do not ignore the cards that are negative instances of the given concept, as they 
provide useful information). 

Since the major goal of the studies was the investigation of the motivational benefits of the 
tutoring feedback strategy, we used a free-choice procedure in the first experiment (i.e., students 
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were totally free to decide the number of tasks they would engage and the amount of time and 
persistence they would invest). Since this free-choice paradigm allowed students to quit the 
learning process after the first task, we were faced with a drop-out problem, particularly for 
students with low and medium levels of self-efficacy. However, students who quit the process after 
the first task did not have enough occasions to benefit from the tutoring feedback strategy. In the 
first study, the tutoring feedback was thus only beneficial for students with high self-efficacy. Thus, 
in the second experiment, students were forced to spend 90 minutes working with the tasks. 
Within these 90 minutes they were however allowed to work at their own pace. Within this 
constraint-choice procedure, the tutoring feedback proved to be beneficial for all students' 
achievement and motivation (see Narciss, 2004, 2006 for further details). These two studies reveal 
that tutoring feedback strategies can only become fully effective for students with low self-efficacy 
if these students do not have the chance to quit the learning process too early. 

b. Bug-related tutoring feedback for written subtraction (Narciss & Huth, 2004, 2006) 

Bug-related feedback refers to all types of feedback messages providing corrective information in 
relation to typical systematic errors (cf. Schimmel, 1988). Based on the ITF-model and on prior 
research on written subtraction (e.g., Brown & VanLehn, 1980; VanLehn, 1990) Narciss and Huth 
(2004; 2006) developed a computer-based training with tutoring feedback strategies for written 
subtraction offering bug-related feedback messages. To do so, we first conducted cognitive task 
and error analyses in order to identify which requirements of written subtraction tasks might be 
considered a source of typical errors. Based on these analyses (a) the scope and functions of the 
tutoring feedback strategy (i.e., helping students to acquire the competencies necessary to solve 
written subtraction tasks, as well as fostering their motivation), (b) the contents of the bug-related 
feedback messages, and (c) formal and technical aspects of presenting the feedback messages 
were specified (see Narciss & Huth, 2004; 2006 for further details). 

The resulting bug-related tutoring feedback strategy provided formative tutoring feedback 
messages with strategically useful information for correcting typical subtraction errors, and 
prompted the learners to apply this information to correct their error in the next trial with the given 
task. It was implemented using a multi-trial schedule with up to three trials: 

After the first trial, learners solving a task incorrectly received knowledge of result (KR) and the 
prompt to try again (i.e., "Sorry, there is an error, try again"). Note that after the first trial the 
error is not immediately flagged, this is done in order to allow the correction of slips, and the self 
identification of errors. Furthermore, KR might be sufficient if learners are able to identify and 
check their own errors. 

After the second trial, learners solving the task incorrectly again, were provided with KR together 
with bug-related strategic information. In cases of a systematic error, location of error is indicated 
through colored marking and a hint concerning the type and source of the error is provided. In 
cases of an unsystematic error, location of error is indicated and a hint to use a worked-out 
example is provided. In both cases learners are offered the possibility to try again. 

After the third trial, if learners failed again to solve the task correctly, they received KR together 
with knowledge on how to proceed in the form of a walk-through worked-out example 
demonstrating step by step how to solve the given task correctly. 

All the feedback messages were provided acoustically by "Subtratino" a pedagogical agent. Error 
flagging (i.e., colored marking of the error location) and hinting information were presented 
visually. To tailor the tutoring feedback levels to the learner's level of competency, the second and 
third levels of feedback were only provided if learners needed it. If learners solved the task 
correctly, they received confirmatory KR (i.e., "Great job! You correctly applied the rule (...)"). After 
students had correctly computed a task, corrected errors, or after they had received the third level 
of feedback, they were offered the opportunity to apply their (newly acquired) knowledge in a 
subsequent task with the same task requirement. Students were provided with a new type of task 
including another task requirement if they had achieved the mastery level, that is, if they 
succeeded in solving two subsequent tasks with the same task requirement on the first trial. 

In two studies with fourth-graders, the effects of this bug-related tutoring feedback strategy on 
students' achievement and motivation were compared to the effects of a widely used computer- 
based feedback strategy providing knowledge of result after a first trial, and knowledge of the 
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correct response after the second trial (KR-KCR-feedback strategy; see Narciss, 2006; Narciss & 
Huth, 2004; 2006 for a detailed description). In order to cope with the evaluation challenges 
outlined above, we selected students with a low level of competency as participants for these 
studies (i.e., students who had solved correctly less than 50% of the pre-test tasks). Furthermore, 
we used a multivariate pre-test - treatment - post-test design, including several achievement 
variables (i.e., pre- and post-test-performance, corrective efficiency, number of task types for 
which mastery level was achieved), as well as domain-specific motivational variables (i.e., 
perceived competence in solving written subtraction tasks). We assumed that the bug-related 
tutoring feedback strategy is more powerful than the KR-KCR-feedback strategy in promoting (a) 
error correction and achievement in the treatment phase, (b) post-test achievement and error 
correction from pre-test to post-test, and (c) students' motivation. The results of both studies 
confirmed these hypotheses (Narciss, 2006; Narciss & Huth, 2006). 

c. Tutoring feedback for fraction tasks - Task with typical-errors paradigm 

The studies summarized so far addressed the methodological problems related to the variability of 
students' error rate and error types by using either learning tasks students had no prior experience 
with (Narciss, 2004), or a selective sample of participants (Narciss & Huth, 2006). Another 
possibility to respond to the methodological complexity raised by the ITF-framework is to 
investigate the effects of tutoring feedback strategies in a task with typical-errors paradigm. 

The core component of such a task with typical-errors paradigm is a set of specifically worked-out 
learning tasks which contain one (or several) specific task-requirements and one (or several) 
typical errors related to these task requirements. The selection of these task-requirements and 
typical errors should be based on cognitive task and error analyses. 

These worked-out tasks with typical-errors (TWTE) can be used to collect data on how students 
with different levels of achievement, motivation, and meta-cognitive skills detect the sources of 
different types of errors and whether they correct these errors with or without different external 
(tutorial) feedback components and strategies. 

Within the TWTE paradigm all students are provided with the same types of errors. Hence, it can 
be used to answer research questions which are particularly important for designing and evaluating 
(adaptive) tutoring feedback types. Such research questions are for example: 

• Which errors have a high probability to be corrected without external tutorial feedback? 

• Which external tutorial feedback components support error correction most effective for 
which kind of errors? 

• How are learner variables, namely prior achievement, motivation, and meta-cognitive skills 
related to the corrective efficiency of the various feedback components? 

• How do learners accept, attend to, and process different feedback components? 

Furthermore, the TWTE paradigm allows for addressing not only generative skills (i.e. cognitive 
skills required for successful task processing) but also meta-cognitive skills (namely assessing 
worked-out task solutions, detecting errors and/or sources of errors, and correcting errors) 
necessary for effective feedback processing (cf. Mathan & Koedinger, 2005). 

Given these potentials of the TWTE-paradigm we used it in our research project "Adaptive tutoring 
Feedback" (AtuFl). The main goal of this project is to develop and evaluate adaptive tutoring 
feedback strategies for fractions tasks provided by the web-based mathematics education system 
ActiveMath. ActiveMath offers access to various mathematical learning objects including a large 
variety of learning tasks and exercises with various types of feedback (for details on ActiveMath 
see Melis, Goguadze, Homik, Libbrecht, Ullrich, & Winterstein, 2006). 


This work was supported by the DFG - Deutsche Forschungsgemeinschaft (Grants ME 1136/8-1 and NA 
738/10-1. 
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In the first phase of the project AtuF we conducted cognitive task and error analyses of fraction 
tasks. These analyses resulted in the development of a domain ontology represented in OWL and in 
the definition of interoperable fraction competencies (Eichelmann, Narciss, Faulhaber, & Melis, 
2008; Melis, Faulhaber, Eichelmann, & Narciss, 2008). In the next step we developed a set of 
interactive fraction tasks explicitly addressing some of the basic fraction competencies, and 
included in each of these tasks one of the typical errors identified through a review of empirical 
fraction error analyses (Eichelmann, et al. 2012). Furthermore, we developed, piloted, and 
iteratively revised a set of tutoring hints and explanations addressing the conceptual and 
procedural knowledge needed to correct these typical errors. These tutoring hints and explanations 
were then combined into multi-trial tutoring feedback strategies providing students with a tutoring 
hint after a first failure, and a tutoring explanation after a second failure. In order to be able to 
track in detail how students use the tutoring feedback for their task processing, we developed a 
special interface - providing students with structured templates that allow the progressive 
solutions of fraction exercises (STEPS-interface, for further details see Eichelmann, Narciss, 
Schnaubert, Melis, & Goguadze, 2011; Narciss, et al., 2013). 

To gain information on (a) the effects of the developed tutoring feedback components and 
strategies, and (b) the relationships among learner variables and these effects, we conducted an 
experimental study with the set of TWTE and the developed tutoring feedback strategies. In a pre- 
test-treatment-post-test control group design, sixth and seventh graders worked in the treatment 
with TWTE-tasks. In the error-correction part of the TWTE, students were asked to replace the 
erroneous step by a correct solution. If their correction failed, they received tutoring feedback and 
were asked to try again. After the third trial or a correct solution, they received a worked-out 
solution and moved on to the next task. 

We conducted detailed logfile-analyses with the data of this study in order to explore the influence 
of content-related properties of feedback (i.e., the type of knowledge communicated by the 
feedback-content: procedural vs. conceptual; and feedback specificity that is the level of 
elaboration of the feedback messages: hint vs. explanation), and learner characteristics (i.e., prior 
competency level, gender, motivational variables) on behavioural process variables; namely 
succeeding vs. failing in correcting the error of a TWTE, and skipping vs. trying to correct an error 
after feedback was provided. Both variables characterize the immediate learner reaction to the 
feedback messages observable in their problem solving activity, and, as such, can serve as 
indicators of the feedback effectiveness. 

In summary, the main findings of these logfile analyses reveal that the impact of pre-assessed 
individual learner characteristics is rather variable across task types and task steps. Gender was 
the only learner characteristic with a relatively stable impact: male students' knowledge gain was 
significantly lower under all feedback conditions, but particularly under feedback strategies 
providing first a conceptual hint. Moreover, after conceptual feedback hints or explanations 
students skipped more frequently, and succeeded less often to correct the error. All other 
differential effects occurred less consistently (Narciss et al., 2013). This result indicates that it is 
important to gain further insights into the dynamic interplay among the individual learner 
characteristics and situational feedback conditions. 

VII. Conclusions 

Using the ITF-model as the theoretical framework for the studies summarized above revealed that 
the ITF-model provides a valuable basis for systematically designing tutoring feedback strategies 
for various learning tasks including concept formation tasks, math tasks, as well as study tasks on 
learning theories (e.g., Narciss, Proske, & Korndle, 2007), and writing assignments (e.g., Strijbos, 
Narciss, Duennebier, 2010). For all the tasks we used so far it was possible to determine rather 
precisely, what constitutes a correct or highly competent solution of these tasks, and which 
competencies are needed to master them. Since, performing task and error analyses for ill- 
structured knowledge domains is difficult and rather costly, if and how the design implications of 
the ITF-model can be applied to the design of feedback strategies for supporting competence 
acquisition in ill-structured knowledge domains is an open issue for further research. 

Furthermore, the application of the design principles derived from the ITF-model in further various 
instructional contexts raises further interesting issues for feedback design and research: 
Concerning computer-based or networked collaborative learning contexts empirical research on the 
effects of formative peer feedback strategies for both the feedback sender and the feedback 
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receiver would be valuable. Moreover, with regard to technology enhanced blended learning 
environments, special interest should be devoted to teacher education and instructional design 
issues such as how to combine human and technical sources of feedback in order to design and 
implement formative multi-source feedback strategies. 

Our studies illuminated some strengths but also limitations of applying the ITF-model to the 
evaluation of tutoring feedback strategies. The ITF-model shows that tutoring feedback messages 
can only have an impact on students' learning if students have occasions to use the feedback 
information for regulating their learning process. Thus, we implemented tutoring feedback 
messages in multi-trial feedback strategies in all our studies. These multi-trial feedback strategies 
provided students with up to three trials to correct errors, and students were free to skip trials and 
work at their own pace. Furthermore, they received the next level of feedback only if they needed 
it. Through the studies conducted with these multi-trial feedback strategies we obtained rich data 
sets which allowed us to investigate at least partly how feedback conditions, feedback properties 
and student's feedback processing may enhance or inhibit the process of competence acquisition. 
Yet, besides the methodological challenges outlined above, we were faced with further challenges, 
including students variability in working pace, number of trials needed to correct an error, as well 
as skipping and drop-out rates. Tackling these evaluation challenges as well as the one's described 
above opens interesting fields of research for interdisciplinary teams of cognitive and educational 
psychologists, instructional scientists and designers, artificial intelligence and computer science 
researchers. 
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